Abnormality detecting apparatus for imaging apparatus

ABSTRACT

An abnormality detecting apparatus for an imaging apparatus that detects the presence of abnormalities in an imaging apparatus mounted on a vehicle to capture vehicle surroundings which includes: a region setting device that sets a region including at least a road boundary portion in an image obtained by the imaging apparatus; a luminance variance value calculating device that calculates a luminance variance value using the luminance value acquired from the image with respect to the region set by the region setting device; and an abnormality detecting device that detects the presence of abnormalities in the imaging apparatus according to the luminance variance value calculated by the luminance variance value calculating device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an abnormality detecting apparatus foran imaging apparatus that detects the presence of abnormal conditions(for example, grime in an optical system, etc.) in the imaging apparatusthat can capture, for example, in the visible light region or infraredregion.

Priority is claimed on Japanese Patent Application No. 2004-347820,filed Nov. 30, 2004, the content of which is incorporated herein byreference.

2. Description of Related Art

There is conventionally known an apparatus that measures the quantity ofreflected light in the light output from the headlights of a vehicleusing a photosensor disposed in the proximity of an infrared cameramounted on the vehicle, and, based on the detected quantity of light,detects the presence of grime on the infrared camera (for example, referto Japanese Unexamined Patent Application, First Publication No.2001-211449).

In the apparatus according to an example of the aforementioned priorart, in addition to the infrared camera, it is necessary to provide thephotosensor as a special apparatus for detecting the presence of grimeon the infrared camera. There is therefore the possibility of theconstitution of the apparatus becoming complicated and the installationposition of the infrared camera being restricted in order to operate thephotosensor in the desired state.

SUMMARY OF THE INVENTION

The present invention was made in view of the above-mentionedcircumstances, and has as its object providing an abnormality detectingapparatus for an imaging apparatus that can easily detect the presenceof abnormal conditions (for example, grime in an optical system, etc.)in an imaging apparatus, while preventing complication of the apparatusconstitution.

In order to attain the object that solves the above-mentioned problem,the present invention provides an abnormality detecting apparatus for animaging apparatus that detects a presence of abnormalities in an imagingapparatus mounted on a vehicle to capture vehicle surroundings,including: a region setting device that sets a region including at leasta road boundary portion in an image obtained by the imaging apparatus; aluminance variance value calculating device that calculates a luminancevariance value using the luminance value acquired from the image withrespect to the region set by the region setting device; and anabnormality detecting device that detects the presence of abnormalitiesin the imaging apparatus according to the luminance variance valuecalculated by the luminance variance value calculating device.

According to the aforementioned abnormality detecting apparatus for animaging apparatus, the road boundary portion on the image obtained byimaging of the imaging apparatus, that is, the region including theboundary between the road that exists in the lower part of the image andthe background that exists in the upper part of the image, is set as aregion to calculate the luminance variance value. It is thereforepossible to improve the detection precision when detecting the presenceof abnormalities in the imaging apparatus in accordance with theluminance variance value and reliability of the detection result. Thatis, when setting a region as a region that contains only the backgroundportion without including the road and boundary portion of the road,regardless of the presence of grime in the imaging apparatus, inaccordance with changes in the travel route (for example, changesbetween a suburban route with relatively few road structures and treesand the like and an urban route with relatively many road structures andtrees and the like), the variance state of the luminance value in theregion (for example, luminance variance value) fluctuates, causingdifficulties in detecting the presence of grime in the imagingapparatus, giving rise to the problem of lower reliability of thedetection accuracy and the detection result. Moreover, when setting aregion as a region that includes only the road without including thebackground portion and the boundary portion of the road, changes in theluminance value in the region become exceedingly low, causingdifficulties in detecting the presence of grime in the imagingapparatus, and so giving rise to the problem of lower reliability of thedetection accuracy and the detection result.

The abnormality detecting apparatus for an imaging apparatus of thepresent invention may further include a region excluding device thatsets an excluded region outside the scope of execution of the process tocalculate the luminance variance value by the luminance variance valuecalculating device in the region set by the region setting device,wherein the luminance variance value calculating device calculates theluminance variance value for the region other than the excluded regionset by the region excluding device among the region set by the regionsetting device.

In this case, by excluding regions determined to be unsuitable whencalculating the luminance variance value of the region, detection of thepresence of abnormalities of the imaging apparatus in accordance withthe luminance variance value can be properly performed. For example,setting as an excluded region a region that may excessively raise theluminance variance value of the region can prevent difficulties indetecting changes in the luminance variance value in accordance with thepresence of grime due to the luminance variance value of the regionexcessively increasing compared to changes in the luminance variancevalue in proportion to the presence of grime in the imaging apparatus.

Moreover, the region excluding device may set as the excluded region aregion in which the luminance value calculated by the luminance valuecalculating device is not less than a specified luminance value.

In this case, setting as an excluded region a region that mayexcessively raise the luminance variance value of the region, that is, aregion in which the luminance value is not less than the specifiedluminance value, can prevent difficulties in detecting changes in theluminance variance value in proportion to the presence of grime by theluminance variance value of the region excessively increasing comparedto changes in the luminance variance value in proportion to the presenceof grime in the imaging apparatus.

The abnormality detecting apparatus for an imaging apparatus of thepresent invention may further include an object extraction device thatextracts an object from the image obtained by the imaging apparatus, andan object type determining device that determines a type of the objectextracted by the object extraction device, wherein the region excludingdevice sets the excluded region based on a determination result of theobject type determining device.

In this case, setting as the excluded region a region including the typeof object that may cause excessive fluctuation in the luminance variancevalue of the region can prevent difficulties in detecting changes in theluminance variance value in proportion to the presence of grime by theluminance variance value of the region excessively fluctuating comparedto changes in the luminance variance value in proportion to the presenceof grime in the imaging apparatus.

The region excluding device may set as the excluded region a region onthe image where an object determined to be a type of object with arelatively high luminance exists in the determination result of theobject type determining device.

In this case, setting as the excluded region the region that may causeexcessive fluctuation in the luminance variance value of the region,that is, the region including the type of object with relatively highluminance such that the luminance value is not less than a predeterminedluminance value, can prevent difficulties in detecting changes in theluminance variance value in proportion to the presence of grime by theluminance variance value of the region excessively increasing comparedto changes in the luminance variance value in proportion to the presenceof grime in the imaging apparatus.

The abnormality detecting apparatus for an imaging apparatus of thepresent invention may further include a luminance variance average valuecalculating device that calculates an average value of a plurality ofthe luminance variance values calculated over a plurality of times bythe luminance variance value calculating device, wherein the abnormalitydetecting apparatus detects the presence of abnormalities in the imagingapparatus in accordance with whether or not the average value calculatedby the luminance variance average value calculating device exceeds apredetermined threshold value.

In this case, determining whether or not the average value of theluminance variance values calculated over a plurality of times exceeds athreshold value can improve the detection precision when detecting thepresence of grime in the imaging apparatus and the reliability of thedetection result regardless of the vehicle circumferential environment,which changes with travel of the vehicle.

The abnormality detecting apparatus for an imaging apparatus of thepresent invention may further include a vehicle status quantitydetection device that detects a status quantity of the vehicle on whichthe imaging apparatus is mounted, wherein the luminance variance averagevalue calculating device sets the average value based only on theluminance variance value calculated by the luminance variance valuecalculating device in the state of the status quantity of the vehicledetected by the vehicle status quantity detection device being aprescribed status quantity.

According to the aforementioned abnormality detecting apparatus for animaging apparatus, setting the average value of the luminance variancevalues based only on the luminance variance value calculated in thestate of the status quantity of the vehicle being a prescribed statusquantity can prevent detection of the presence of grime in the imagingapparatus based on an improper luminance variance value caused by thestate of the vehicle. For example, by setting a state in which thevehicle speed is not less than a prescribed speed as the prescribedstatus quantity with regard to the vehicle status quantity, the distancebetween the vehicle and a preceding vehicle or another object is notless than a predetermined distance, thereby preventing the size of asingle object on the image from becoming excessively large, and therebythe share of the luminance variance value according to the single objectincreasing excessively with respect to the luminance variance value ofthe region. Also, by setting, for example, the state of the absolutevalue of the yaw rate to be a prescribed threshold value or less or thestate of no brake operation by the driver as the prescribed statusquantity for the vehicle status quantity, the average value of theluminance variance value can be a suitable value in a state in which thetraveling behavior of the vehicle is relatively stable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the constitution of the abnormalitydetecting apparatus for an imaging apparatus according to an embodimentof the present invention.

FIG. 2 is a diagram showing a vehicle equipped with the abnormalitydetecting apparatus for an imaging apparatus shown in FIG. 1.

FIG. 3 is a lineblock diagram of the image processing unit shown in FIG.1.

FIG. 4 is diagram showing an example of a determination region set on aninfrared image.

FIG. 5 is a diagram showing examples of excluded regions set within thedetermination region shown in FIG. 4.

FIG. 6 is a diagram showing an example of the change in an infraredimage and luminance histogram depending on the presence/absence ofgrime.

FIG. 7 is a flowchart showing the operation of the abnormality detectingapparatus for an imaging apparatus shown in FIG. 1.

FIG. 8 is a lineblock diagram of the image processing unit according toa modification of the present embodiment.

FIG. 9 is a flowchart showing the operation of the abnormality detectingapparatus for an imaging apparatus according to the modification of thepresent embodiment.

FIG. 10 is a flowchart showing the operation of the abnormalitydetecting apparatus for an imaging apparatus according to themodification of the present embodiment.

FIG. 11 is a flowchart showing the operation of the abnormalitydetecting apparatus for an imaging apparatus according to themodification of the present embodiment.

FIG. 12 is a flowchart showing the operation of the abnormalitydetecting apparatus for an imaging apparatus according to themodification of the present embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Hereunder, an abnormality detecting apparatus for an imaging apparatusaccording to one embodiment of the present invention is described withreference to the drawings.

The abnormality detecting apparatus for an imaging apparatus accordingto the present embodiment, for example as shown in FIG. 1, includes: animage processing unit 1 equipped with a CPU (Central Processing Unit)that controls the abnormality detecting apparatus for an imagingapparatus; two infrared cameras 2R and 2L that are capable of detectingdistant infrared radiation; a yaw rate sensor 3 that detects the yawrate of the vehicle; a vehicle speed sensor 4 that measures thetraveling speed of the vehicle; a brake sensor 5 that detects a driver'sbraking operation; a loudspeaker 6; and a display apparatus 7. Forexample, the image processing unit 1 detects a moving object such as apedestrian or an animal in front of the vehicle in its travelingdirection from infrared images of the vehicle surroundings that arecaptured by the two infrared cameras 2R and 2L, and from detectionsignals relating to the traveling status of the vehicle that aredetected by each of the sensors 3, 4, and 5. In the case where thepossibility of a collision between the detected moving object and thevehicle is determined, a warning is output via the loudspeaker 6 or thedisplay apparatus 7.

Moreover, the display apparatus 7 is, for example, constructed includinga display apparatus integrated with gauges that display varioustraveling states of the vehicle, a display apparatus such as anavigation device, and furthermore an HUD (Head Up Display) 7 a thatdisplays various information at a position on the front window where thefield of front vision of the driver is not impaired.

In addition, the image processing unit 1 includes an A/D converter, thatconverts input analog signals to digital signals, an image memory thatstores digitized image signals (luminance values), a CPU (centralprocessing unit) that performs various arithmetic processing, a RAM(Random Access Memory) that is used for storing data in the middle ofthe arithmetic processing, a ROM (Read Only Memory) that stores programsthat are performed by the CPU and tables, maps and the like, and anoutput circuit that outputs drive signals for the loudspeaker 6 anddisplay signals for the HUD 7 a. The image processing unit 1 isconstructed such that the output signals of the infrared cameras 2R and2L, and the respective sensors, 3, 4, and 5 are input into the CPU afterbeing converted to digital signals.

Furthermore, as shown in FIG. 2, two infrared cameras 2R and 2L aredisposed at the front of the vehicle 10 at positions symmetrical in thewidth direction relative to the central axis of the vehicle 10. Theoptical axes of both cameras 2R and 2L are parallel to each other, andboth infrared cameras 2R and 2L are secured at the same height from theroad surface. A characteristic of the infrared cameras 2R and 2L is thatthe output signal level (that is, luminance) increases as thetemperature of the object increases.

Moreover, the HUD 7 a is provided so as to display the images at aposition on the front window of the vehicle 10, where the field of frontvision of the driver is not impaired.

The image processing unit 1 is, as for example shown in FIG. 3, equippedwith an image memory 11, a determination region setting portion 12, avehicle behavior estimating portion 13, an excluded region settingportion 14, a luminance variance calculating portion 16 and a cameragrime determination portion 17. In particular, the vehicle behaviorestimating portion 13 of the image processing unit 1 has a yaw ratesensor 3, a vehicle speed sensor 4, and a brake sensor 5, as well as anacceleration sensor and a windshield wiper switch (windshield wiper SW).The detection signal output from a vehicle status quantity sensor 21that detects various kinds of vehicle status quantities is input, and aninforming apparatus 22 that has a loudspeaker 6 and a display apparatus7 is connected to the camera grime determination portion 17 of the imageprocessing unit 1.

In this image processing unit 1, the image memory 11 stores infraredimages received from the infrared camera 2R (or 2L) as digital data.

The determination region setting portion 12 sets a determination regionon the infrared image based on the optical design (for example, cameraangle of view) and installation direction of the infrared camera 2R (or2L) established in advance, and as required, corrects the determinationregion that is set based on the estimation result of the vehiclebehavior estimating portion 13, for example, changes in the optical axiscaused by displacement in the installation position of the infraredcamera 2R (or 2L).

As shown in FIG. 4, this determination region is a region that containsat least a road R, a boundary portion of the road BR and a backgroundportion B. Here, if the region which contains only the backgroundportion B upward on the infrared image without the road R and theboundary portion of the road BR is set as the determination region,regardless of the presence of grime on the infrared camera 2R (or 2L),in accordance with changes in the travel route (for example, changesbetween a suburban route with relatively few road structures and treesand the like and an urban route with relatively many road structures andtrees and the like), the variance state of the luminance value in theregion (for example, luminance variance value) fluctuates, and executionof grime determination that is described later becomes difficult, givingrise to the problem of the detection accuracy of grime determination andthe reliability of the detection result dropping.

Also, if a region that includes only the road R in the lower portion ofthe infrared image without including the background portion B and theboundary portion of the road BR is set as the determination region,changes in the luminance value in the determination region becomeexceedingly small, and execution of grime determination, which isdescribed later, becomes difficult, giving rise to the problem of thedetection accuracy of grime determination and the reliability of thedetection result dropping.

Based on the detection signal output from a vehicle status quantitysensor 21, the vehicle behavior estimating portion 13 calculates, asdisplacement in the installation position and installation direction ofthe infrared camera 2R (or 2L) set in advance, the pan angle indicating,for example, the displacement in the horizontal direction of the opticalaxis of the infrared camera 2R (or 2L) with respect to the vehicletraveling direction, or the pitch angle indicating, for example,displacement in the vertical direction of the optical axis of theinfrared camera 2R (or 2L) with respect to the vehicle travelingdirection that occurs due to pitching of the vehicle.

For example, the vehicle behavior estimating portion 13 calculates anapproximated straight line LMV that approximates the relative movementtrack of a static object detected on an infrared image. With respect tothe camera coordinate system (Xc, Yc, Zc) based on the optical axis ofthe infrared camera 2R (or 2L), the angle formed between a straight lineformed by the approximated straight line LMV projected on the flat planeXc-Zc and the Zc axis serves as the pan angle, and the angle formedbetween the straight line formed by the approximated straight line LMVprojected on the flat plane Yc-Zc and the Zc axis serves as the pitchangle.

In addition, the camera coordinate system (Xc, Yc, Zc) is calculated asshown in the numerical expression (1), with the center position of themounting position of the infrared cameras 2R and 2L in front of avehicle 10 serving as the origin O, and the coordinates on the infraredimage set so that the center of the infrared image is the origin, thehorizontal direction is the x direction, and the vertical direction isthe y direction. Furthermore, the coordinates (xc, yc) are thecoordinates that have been converted from the coordinates (x, y) in thereference image (for example, the right image) into the coordinates in avirtual image obtained by aligning the origin O of the real space andthe center of the image data so that they coincide, based on therelative position relationship between the mounting position of theinfrared camera 2R and the origin O of the real space. Also, in thenumerical expression (1) below, the distance between the vehicle 10 andthe object, that is, the distance z (m), (object distance) from thelenses of the infrared cameras 2R and 2L to the object, is for examplecalculated based on the base length of the cameras, that is thehorizontal distance D (m) between center positions of each imagingdevice of the infrared cameras 2R and 2L, the focus distance of thecamera, that is, the focus distance f (m) of each lens of the infraredcameras 2R and 2L, the pixel pitch p (m/pixel), and parallax Δd (pixel)at pixel level. $\begin{matrix} \begin{matrix}{\begin{bmatrix}{Xc} \\{Yc} \\{Zc}\end{bmatrix} = \begin{bmatrix}{{xc} \times {z/F}} \\{{yc} \times {z/F}} \\{z}\end{bmatrix}} \\{z = \frac{f \times D}{\Delta\quad d \times p}} \\{F = \frac{f}{p}}\end{matrix} \} & (1)\end{matrix}$

Within the determination region set by the determination region settingportion 12, the excluded region setting portion 14 excludes from thegrime determination target region, which is described later, regions ofhigh luminance in which the luminance value is not less than a specifiedluminance value set in accordance with the luminance gain or luminancelevel of the infrared camera 2R (or 2L) set in advance or suitablychanged.

For example, as shown in FIG. 5, the excluded region setting portion 14sets as excluded regions those regions OB1, . . . , OB6 having aluminance value not less than a predetermined luminance within thedetermination region on the infrared image.

Moreover, in the case of a specified heating element detected by anobject determination portion 15 (for example, an exhaust pipe or vendingmachine and the like) existing within the determination region on theinfrared image, the excluded region setting portion 14 sets the regioncontaining this heating element as an excluded region.

For the region other than the excluded regions set by the excludedregion setting portion 14 within the determination region set by thedetermination region setting portion 12, the luminance variancecalculating portion 16 first calculates a luminance histogram that is afrequency variance with respect to luminance value, and then calculatesthe variance (luminance variance) of the luminance histogram.

The camera grime determination portion 17 determines whether or notgrime was generated in the infrared camera 2R (or 2L) based on theaverage value of a plurality of luminance variances (luminance varianceaverage value) calculated by the luminance variance calculating portion16 over a specified time for example.

For example, the camera grime determination portion 17 determineswhether or not the calculated luminance variance average is less than aspecified variance threshold (for example, a value which has hysteresis)set in accordance with the luminance gain or luminance level of theinfrared camera 2R (or 2L) set in advance or suitably changed. And whenthis determination result is “YES”, as shown for example in FIG. 6, itis determined that the contrast (light-and-darkness difference) of theinfrared image has dropped and the luminance variance value has fallendue to the occurrence of grime in the optical system of the infraredcamera 2R (or 2L), and so grime is determined as existing in theinfrared camera 2R (or 2L). Meanwhile, when this determination result is“NO”, it is determined that there is no grime in the infrared camera 2R(or 2L).

In addition, when calculating the luminance variance average value, thecamera grime determination portion 17, based on the detection signaloutput from the vehicle status quantity sensor 21, selects a suitableluminance variance from the plurality of luminance variances, beingenabled to calculate the luminance variance average value solely fromthe selected luminance variance.

For example, by selecting the luminance variance calculated in a statein which the traveling behavior of the vehicle is relatively stable, orselecting the luminance variance calculated in a state in which thevehicle speed is not less than a specified speed, the camera grimedetermination portion 17 can curb decreases in the luminance variancecaused by, for example, the distance between the vehicle and a precedingvehicle or a structure becoming relatively short in the state of arelatively low vehicle speed, thereby causing the area of the singlepreceding vehicle or object to increase excessively on the infraredimage.

When it is determined by the camera grime determination portion 17 thatgrime is present in the infrared camera 2R (or 2L), the informingapparatus 22 outputs an audible warning such as an alarm sound or alarmvoice via the loudspeaker 6 or a visual warning such as a display viathe display apparatus 7.

In addition, the camera grime determination portion 17 determineswhether or not there is the state of determination that grime exists inthe infrared camera 2R (or 2L) caused by the weather, such as a rainfalland snowfall, based on the detection signal according to ON/OFF ofactuation of the windshield wiper output from the windshield wiperswitch (windshield wiper SW) of the vehicle status quantity sensor 21.When this determination result is “YES”, it alters the informationcontent from the informing apparatus 22 or temporarily suspends theprocess that determines the presence of grime.

The abnormality detecting apparatus for an imaging apparatus accordingto the present embodiment is provided with the construction describedabove. Next, the operation of the abnormality detecting apparatus for animaging apparatus, particularly the process to determine the presence ofgrime in the infrared camera 2R (or 2L), is described with reference tothe drawings.

First of all, in step S01 shown in FIG. 7, an infrared image is obtainedfrom the infrared camera 2R (or 2L).

Next, in step S02, a determination region including at least the road R,the boundary portion of the road BR and the background portion B is seton the obtained infrared images.

Next, in step S03, high luminance regions in which the luminance valueis not less than a prescribed luminance value are set as excludedregions within the determination region that is set.

Next, in step S04, it is determined whether a vehicle speed V measuredby the vehicle speed sensor 4 is not less than a specified thresholdspeed Veth.

When this determination result is “NO”, the processing is terminated.

On the other hand, when the determination result is “YES”, the flowproceeds to step S05.

In step S05, it is determined whether or not the absolute value of theyaw rate co detected by the yaw rate sensor 3 is not higher than aspecified threshold value Yawth.

When the determination result of step S05 is “NO”, the processing isterminated.

On the other hand, when the determination result of step S05 is “YES”,the flow proceeds to step S06.

Then, in step S06, it is determined whether or not the state of thebrake SW, which depends on a driver's braking operation detected by thebrake sensor 5, is OFF.

When this determination result is “NO”, the processing is terminated.

On the other hand, when the determination result is “YES”, the flowproceeds to step S07.

Then, in step S07, it is determined whether or not the windshield wiperSW is OFF.

When this determination result in step S07 is “NO”, the processing isterminated.

On the other hand, when the determination result in step S07 is “YES”,the flow proceeds to step S08.

Then, in step S08, a luminance histogram is calculated for the regionother than the excluded regions within the determination region, and thevariance of the luminance histogram (luminance variance) is calculated.

Next, in step S09, the average value of a plurality of luminancevariances (luminance variance average value) calculated over a specifiedtime is calculated.

Next, in step S10, it is determined whether or not it was determined inthe previous process whether there is no grime in the infrared camera 2R(or 2L).

When this determination result is “NO”, the flow proceeds to step S13that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S11.

In step S11, it is determined whether or not the luminance varianceaverage value is less than a specified lower limit variance thresholdvalue VL set in accordance with the luminance gain or luminance level ofthe infrared camera 2R (or 2L) set in advance or suitably changed.

When this determination result is “NO”, the processing is terminated.

On the other hand, when the determination result is “YES”, the flowproceeds to step S12.

In step S12, as the grime determination result it is set that there isgrime, and the flow proceeds to step S15 that is described later.

Also, in step S13, it is determined whether or not the luminancevariance average value is not less than a predetermined upper limitvariance threshold value VU set in accordance with the luminance gain orluminance level of the infrared camera 2R (or 2L) set in advance orsuitably changed.

When this determination result is “NO”, the processing is terminated.

On the other hand, when the determination result is “YES”, the flowproceeds to step S14.

In step S14, as the grime determination result it is set that there isno grime, and the flow proceeds to step S15.

In step S15, the grime determination result is reported to the driver ofthe vehicle via the informing apparatus 22, and the processing isterminated.

As described above, the abnormality detecting apparatus for an imagingapparatus according to the present embodiment sets a region on theinfrared image that includes at least a road R, a boundary portion ofthe road BR and a background portion B as the determination region, forwhich the luminance variance value is calculated. Therefore, thedetection accuracy when detecting the presence of grime on the infraredcameras 2R and 2L according to the luminance variance value and thereliability of the detection result can be improved.

Moreover, since regions that may excessively raise the luminancevariance value of the region, that is, regions whose luminance value isnot less than a specified luminance value, or regions that includeobjects of a type that may cause excessive fluctuation in the luminancevariance value of the region (for example, an exhaust pipe of anothervehicle or a vending machine and the like) can be set as excludedregions, detection of the presence of grime on the infrared cameras 2Rand 2L according to the luminance variance value can be properlyperformed.

Also, by determining whether or not the average value of luminancevariances values calculated over a plurality of times exceeds a certainthreshold (the lower limit variance threshold value VL or the upperlimit variance threshold value VU) (that is, whether it is less than thelower limit variance threshold value VL or not less than the upper limitvariance threshold value VU), the detection accuracy when detecting thepresence of grime on the infrared cameras 2R and 2L and the reliabilityof the detection result can be improved regardless of the vehiclesurroundings environment which changes with travel of the vehicle.

Moreover, by calculating the average value of luminance variance valuesbased only on the luminance variance values calculated in the state ofthe status quantity of the vehicle being a certain status quantity (forexample, the vehicle velocity V being not less than a specifiedthreshold velocity Veth, the yaw rate ω being not higher than aspecified threshold value Yawth, and the brake SW being OFF), detectionof the presence of grime on the infrared cameras 2R and 2L based on animproper luminance variance value caused by the state of the vehicle canbe prevented.

In addition, in the embodiment described above, although the presence ofgrime on the infrared camera 2R (or 2L) was determined according towhether or not a luminance variance average value is less than aspecified variance threshold, it is not limited thereto. For example, itmay be determined whether or not the difference between a specifiedreference luminance variance value (for example, a luminance varianceaverage value) and a luminance variance calculated by the luminancevariance calculating portion 16 is less than a specified thresholddifference set in accordance with the luminance gain or luminance levelof the infrared camera 2R (or 2L) set in advance or suitably changed.When this determination result is “NO”, then it is determined that grimeexists on the infrared camera 2R (or 2L), and on the other hand when thedetermination result is “YES”, it is determined that there is no grimeon the infrared camera 2R (or 2L).

In addition, in the embodiment described above, although grimedetermination was carried out independently for each infrared camera 2Rand 2L, it is not limited thereto. For example, grime determination maybe performed based on the determination result for the infrared camera2R (hereafter referred to as the first infrared camera 2R) and thedetermination result for the infrared camera 2L (hereafter referred toas the second infrared camera 2L).

In this modification, as shown for example in FIG. 8, the imageprocessing unit 1 is equipped with, corresponding to the first andsecond infrared cameras 2R and 2L, image memories 11R and 11L,determination region setting portions 12R and 12L, a vehicle behaviorestimating portion 13, excluded region setting portions 14R and 14L, anobject determination portion 15, luminance variance calculating portions16R and 16L, camera grime determination portions 17R and 17L, a cameragrime comparison determination portion 18, and a weather determinationportion 19. In particular, the vehicle behavior estimating portion 13 ofthe image processing unit 1 has a yaw rate sensor 3, a vehicle speedsensor 4, and a brake sensor 5, as well as an acceleration sensor and awindshield wiper switch (windshield wiper SW). The detection signaloutput from the vehicle status quantity sensor 21 that detects variouskinds of vehicle status quantities is input, and an informing apparatus22 that has a loudspeaker 6 and a display apparatus 7 is connected tothe camera grime determination portion 17 of the image processing unit1.

In this image processing unit 1, each image memory 11R and 11L storesinfrared images received from each infrared camera 2R and 2L as digitaldata.

Each determination region setting portion 12R and 12L sets adetermination region on the infrared image based on the optical design(for example, camera angle of view) and installation direction of eachinfrared camera 2R and 2L established in advance, and as required,corrects the determination region that is set based on the estimationresult of the vehicle behavior estimating portion 13, for example,changes in the optical axis caused by displacement in the installationposition of the infrared cameras 2R and 2L.

Based on the detection signal output from a vehicle status quantitysensor 21, for each infrared camera 2R and 2L the vehicle behaviorestimating portion 13 calculates, as displacement in the installationposition and installation direction of the infrared cameras 2R and 2Lset in advance, the pan angle that shows, for example, the displacementin the horizontal direction of the optical axis of the infrared cameras2R and 2L with respect to the vehicle traveling direction, or the pitchangle that shows, for example, displacement in the vertical direction ofthe optical axis of the infrared cameras 2R and 2L with respect to thevehicle traveling direction that occurs due to pitching of the vehicle.

For example, the vehicle behavior estimating portion 13 calculates anapproximated straight line LMV that approximates the relative movementtrack of a static object detected on an infrared image. With respect tothe camera coordinate system (Xc, Yc, Zc) based on the optical axis ofeach infrared camera 2R and 2L, the angle formed between a straight lineformed by the approximated straight line LMV projected on the flat planeXc-Zc and the Zc axis serves as the pan angle, and the angle formedbetween the straight line formed by the approximated straight line LMVprojected on the flat plane Yc-Zc and the Zc axis serves as the pitchangle.

In addition, the camera coordinate system (Xc, Yc, Zc) is calculated asshown in the numerical expression (1) above, with the center position ofthe mounting position of the infrared cameras 2R and 2L in front of avehicle 10 serving as the origin O, and the coordinates on the infraredimage set so that the center of the infrared image is the origin, thehorizontal direction is the x direction, and the vertical direction isthe y direction. Furthermore, the coordinates (xc, yc) are thecoordinates that have been converted from the coordinates (x, y) in thereference image (for example, the right image) into the coordinates in avirtual image obtained by aligning the origin O of the real space andthe center of the image data so that they coincide, based on therelative position relationship between the mounting position of theinfrared camera 2R and the origin O of the real space. Also, in thenumerical expression (1) above, the distance between the vehicle 10 andthe object, that is, the distance z (m) (object distance) from thelenses of the infrared cameras 2R and 2L to the object, is for examplecalculated based on the base length of the cameras, that is thehorizontal distance D (m) between center positions of each imagingdevice of the infrared cameras 2R and 2L, the focus distance of thecamera, that is, the focus distance f (m) of each lens of the infraredcameras 2R and 2L, the pixel pitch p (m/pixel), and parallax Δd (pixel)at pixel level.

Within each determination region set by the determination region settingportions 12R and 12L, the excluded region setting portions 14R and 14Lexclude from the grime determination target region, which is describedlater, regions of high luminance in which the luminance value is notless than a specified luminance value set in accordance with theluminance gain or luminance level of the infrared cameras 2R and 2L setin advance or suitably changed.

Moreover, in the case of a specified heating element detected by theobject determination portion 15 (for example, an exhaust pipe of anothervehicle or a vending machine and the like) existing within thedetermination region on the infrared image, the excluded region settingportions 14R and 14L set the region containing this heating element asan excluded region.

For the region other than the excluded regions set by the excludedregion setting portions 14R and 14L within each determination region setby the determination region setting portions 12R and 12L, the luminancevariance calculating portions 16R and 16L first calculate a luminancehistogram that is a frequency variance with respect to luminance value,and then calculate the variance (luminance variance) of the luminancehistogram.

Each camera grime determination portion 17R and 17L determines whetheror not grime was generated in the infrared cameras 2R and 2L based onthe average value of a plurality of luminance variances (luminancevariance average value) calculated by the luminance variance calculatingportions 16R and 16L over a specified time for example.

For example, each camera grime determination portion 17R and 17Ldetermines whether or not the calculated luminance variance average isless than a specified variance threshold (for example, a value which hashysteresis) set in accordance with the luminance gain or luminance levelof the infrared cameras 2R and 2L set in advance or suitably changed.And when this determination result is “YES”, it is determined that thecontrast (light-and-darkness difference) of the infrared image hasdropped and, accordingly, the luminance variance value has fallen due tothe occurrence of grime in the optical system of the infrared cameras 2Rand 2L, and so as the grime determination result of the infrared cameras2R and 2L it is determined that grime exists. Meanwhile, when thisdetermination result is “NO”, as the grime determination result of theinfrared cameras 2R and 2L it is determined that there is no grime.

In addition, when calculating the luminance variance average value, thecamera grime determination portions 17R and 17L, based on the detectionsignal output from the vehicle status quantity sensor 21, selects asuitable luminance variance from the plurality of luminance variances,being enabled to calculate the luminance variance average value solelyfrom the selected luminance variance.

For example, by selecting the luminance variance calculated in a statein which the traveling behavior of the vehicle is relatively stable, orselecting the luminance variance calculated in a state in which thevehicle speed is not less than a specified speed, the camera grimedetermination portions 17R and 17L can curb decreases in the luminancevariance caused by, for example, the distance between the vehicle and apreceding vehicle or a structure becoming relatively short in the stateof a relatively low vehicle speed, thereby causing the area of thesingle preceding vehicle or object to increase excessively on theinfrared image.

The camera grime comparison determination portion 18 determines whetheror not grime has occurred at the infrared cameras 2R and 2L by comparingthe luminance variance average values calculated for each infraredcamera 2R and 2L.

For example, the camera grime comparison determination portion 18determines whether or not the difference between the luminance varianceaverage value of the first infrared camera 2R and the luminance varianceaverage value of the second infrared camera 2L is less than a specifiedvariance threshold (for example, a value which has hysteresis) set inaccordance with the luminance gain or luminance level of each infraredcamera 2R and 2L set in advance or suitably changed. And when thisdetermination result is “YES”, due to the occurrence of grime in theoptical system of the first or second infrared cameras 2R and 2L, it isdetermined that the contrast (light-and-darkness difference) of oneinfrared image has relatively dropped compared to the contrast of theother infrared image, and consequently that the difference between theluminance variance average values has increased. Therefore, as the grimecomparison determination result, grime is determined to exist on one ofthe first or second infrared cameras 2R and 2L and not exist on theother.

When it is determined by the camera grime determination portion 17 thatgrime is present in the infrared cameras 2R and 2L, or when it isdetermined by the camera grime comparison determination portion 18 thatgrime is present on either one of the first infrared camera 2R or thesecond infrared cameras 2L, the informing apparatus 22 outputs anaudible warning such as an alarm sound or alarm voice via theloudspeaker 6 or a visual warning such as a display via the displayapparatus 7.

In addition, each camera grime determination portion 17R and 17L and thecamera grime comparison determination portion 18 determines whether ornot there is the state of determination that grime exists in theinfrared cameras 2R and 2L in accordance with the determination resultof the weather determination portion 19, for example the determinationresult for the state of the weather such as a rainfall and snowfallbased on the detection signal according to ON/OFF of actuation of thewindshield wiper output from the windshield wiper switch (windshieldwiper SW) of the vehicle status quantity sensor 21. When thisdetermination result is “YES”, it alters the information content fromthe informing apparatus 22 or temporarily suspends the process thatdetermines the presence of grime.

The abnormality detecting apparatus for an imaging apparatus accordingto the present modification is provided with the construction describedabove. Next, the operation of the abnormality detecting apparatus for animaging apparatus, particularly the process to determine the presence ofgrime in each infrared camera 2R and 2L, is described with reference tothe drawings.

First of all, in step S21 shown in FIG. 9, an infrared image is obtainedfrom each infrared camera 2R and 2L.

Next, in step S22, a determination region including at least the road R,the boundary portion of the road BR and the background portion B is seton each obtained infrared image.

Next, in step S23, high luminance regions in which the luminance valueis not less than a prescribed luminance value are set as excludedregions within the determination region.

Next, in step S24, a luminance histogram is calculated for the regionother than the excluded regions within each determination region, andthe variance of the luminance histogram (luminance variance) iscalculated.

Next, in step S25, the average value of a plurality of luminancevariances (luminance variance average value) calculated over a specifiedtime is calculated for each infrared camera 2R and 2L.

Next, in step S26, it is determined whether or not it was determined inthe previous process that there is no grime, as the grime determinationresult for the first infrared camera 2R.

When this determination result is “NO”, the flow proceeds to step S29that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S27.

In step S27, it is determined whether or not the luminance varianceaverage value of the first infrared camera 2R is less than a specifiedlower limit variance threshold value VL1 set in accordance with theluminance gain or luminance level of the first infrared camera 2R set inadvance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S31that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S28.

In step S28, the setting is made that grime exists as the grimedetermination result of the first infrared camera 2R, and the flowproceeds to step S31 described later.

In step S29, it is determined whether or not the luminance varianceaverage value of the first infrared camera 2R is not less than aspecified upper limit variance threshold value VU1 set in accordancewith the luminance gain or luminance level of the first infrared camera2R set in advance or suitably changed.

When this determination result is “NO”, flow proceeds to step S31described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S30.

In step S30, the setting is made that grime does not exist as the grimedetermination result of the first infrared camera 2R, and the flowproceeds to step S31.

In step S31, it is determined whether or not it was determined thatthere is no grime in the first infrared camera 2R as the grimecomparison determination result in the previous process.

When this determination result is “NO”, flow proceeds to step S35described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S32.

In step S32, it is determined whether or not the luminance varianceaverage value of the first infrared camera 2R is less than a specifiedlower limit variance threshold value VL2 set in accordance with theluminance gain or luminance level of the first infrared camera 2R set inadvance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S38that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S33.

In step S33, it is determined whether or not the value obtained bysubtracting the luminance variance average value of the first infraredcamera 2R from the luminance variance average value of the secondinfrared camera 2L is not less than a specified upper limit differencethreshold value Vdmax set in accordance with the luminance gain orluminance level of each infrared camera 2R and 2L set in advance orsuitably changed.

When this determination result is “NO”, the flow proceeds to step S38that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S34.

In step S34, the setting is made that grime exists as the grimecomparison determination result of the first infrared camera 2R, and theflow proceeds to step S38 that is described later.

In step S35, it is determined whether or not the value obtained bysubtracting the luminance variance average value of the first infraredcamera 2R from the luminance variance average value of the secondinfrared camera 2L is less than a specified lower limit differencethreshold value Vdmin set in accordance with the luminance gain orluminance level of each infrared camera 2R and 2L set in advance orsuitably changed.

When this determination result is “NO”, the flow proceeds to step S37that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S36.

In step S36, the setting is made that grime does not exist as the grimecomparison determination result of the first infrared camera 2R, and theflow proceeds to step S38 that is described later.

In step S37, it is determined whether or not the luminance varianceaverage value of the first infrared camera 2R is not less than aspecified upper limit variance threshold value VU2 set in accordancewith the luminance gain or luminance level of the first infrared camera2R set in advance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S38that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to the aforementioned step S36.

Next, in step S38, it is determined whether or not it was determined inthe previous process that there is no grime, as the grime determinationresult for the second infrared camera 2L.

When this determination result is “NO”, the flow proceeds to step S41that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S39.

In step S39, it is determined whether or not the luminance varianceaverage value of the second infrared camera 2L is less than a specifiedlower limit variance threshold value VL1 set in accordance with theluminance gain or luminance level of the second infrared camera 2L setin advance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S43that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S40.

In step S40, the setting is made that grime exists as the grimedetermination result of the second infrared camera 2L, and the flowproceeds to step S43 that is described later.

In step S41, it is determined whether or not the luminance varianceaverage value of the second infrared camera 2L is not less than aspecified upper limit variance threshold value VU1 set in accordancewith the luminance gain or luminance level of the second infrared camera2L set in advance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S43that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S42.

In step S42, the setting is made that grime does not exist as the grimedetermination result of the second infrared camera 2L, and the flowproceeds to step S43.

In step S43, it is determined whether or not it was determined thatthere is no grime in the second infrared camera 2L as the grimecomparison determination result in the previous process.

When this determination result is “NO”, flow proceeds to step S47described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S44.

In step S44, it is determined whether or not the luminance varianceaverage value of the second infrared camera 2L is less than a specifiedlower limit variance threshold value VL2 set in accordance with theluminance gain or luminance level of the second infrared camera 2L setin advance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S50that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S45.

In step S45, it is determined whether or not the value obtained bysubtracting the luminance variance average value of the second infraredcamera 2L from the luminance variance average value of the firstinfrared camera 2R is not less than a specified upper limit differencethreshold value Vdmax set in accordance with the luminance gain orluminance level of each infrared camera 2R and 2L set in advance orsuitably changed.

When this determination result is “NO”, the flow proceeds to step S50that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S46.

In step S46, the setting is made that grime exists as the grimecomparison determination result of the second infrared camera 2L, andthe flow proceeds to step S50 that is described later.

In step S47, it is determined whether or not the value obtained bysubtracting the luminance variance average value of the second infraredcamera 2L from the luminance variance average value of the firstinfrared camera 2R is less than a specified lower limit differencethreshold value Vdmin set in accordance with the luminance gain orluminance level of each infrared camera 2R and 2L set in advance orsuitably changed.

When this determination result is “NO”, the flow proceeds to step S49that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to step S48.

In step S48, the setting is made that grime does not exist as the grimecomparison determination result of the second infrared camera 2L, andthe flow proceeds to step S50 that is described later.

In step S49, it is determined whether or not the luminance varianceaverage value of the second infrared camera 2L is not less than aspecified upper limit variance threshold value VU2 set in accordancewith the luminance gain or luminance level of the second infrared camera2L set in advance or suitably changed.

When this determination result is “NO”, the flow proceeds to step S50that is described later.

On the other hand, when the determination result is “YES”, the flowproceeds to the aforementioned step S48.

In step S50, it is determined whether or not it was determined thatgrime exists by at least any determination result among the grimedetermination result of the first infrared camera 2R, the grimecomparison determination result of the first infrared camera 2R, thegrime determination result of the second infrared camera 2L, and thegrime comparison determination result of the second infrared camera 2L.

When the determination result is “YES”, the flow proceeds to step S51.In this step S51, the setting is made for the grime determination resultthat grime exists, that is, there is grime for at least one of theinfrared cameras 2R and 2L, and the flow proceeds to step S54 describedlater.

On the other hand, when the determination result is “NO”, the flowproceeds to step S52.

In step S52, it is determined whether or not it was determined thatthere is no grime according to all the determination results of thegrime determination result of the first infrared camera 2R, the grimecomparison determination result of the first infrared camera 2R, thegrime determination result of the second infrared camera 2L, and thegrime comparison determination result of the second infrared camera 2L.

When the determination result is “YES”, the flow proceeds to step S53.In this step S53, the setting is made for the grime determination resultthat there is no grime, that is, there is no grime in both of theinfrared cameras 2R and 2L, and the flow proceeds to step S54 describedlater.

On the other hand, when this determination result is “NO”, theprocessing is terminated.

In step S54, the grime determination result is reported to the driver ofthe vehicle via the informing apparatus 22, and the processing isterminated.

As stated above, the aforementioned abnormality detecting apparatus foran imaging apparatus of this modification can relatively detect thepresence of abnormalities in each infrared camera 2R and 2L regardlessof the setting state of each infrared camera 2R and 2L. Moreover, sincethe presence of abnormalities is detected in the imaging apparatus inaccordance with the difference between luminance variance average valuesof images, in addition to the luminance variance average values ofimages, for example, even in the case that the difference between theluminance variance average values of images is not above a specifiedvalue, and the corresponding infrared cameras 2R and 2L are determinedto be normal, in the case that the luminance variance average values ofthe images are less than a specified value, it can determine that thecorresponding infrared cameras 2R and 2L are in an abnormal state.

While preferred embodiments of the invention have been described andillustrated above, it should be understood that these are exemplary ofthe invention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

1. An abnormality detecting apparatus for an imaging apparatus thatdetects a presence of abnormalities in an imaging apparatus mounted on avehicle to capture vehicle surroundings, comprising: a region settingdevice that sets a region including at least a road boundary portion inan image obtained by the imaging apparatus; a luminance variance valuecalculating device that calculates a luminance variance value using theluminance value acquired from the image with respect to the region setby the region setting device; and an abnormality detecting device thatdetects the presence of abnormalities in the imaging apparatus accordingto the luminance variance value calculated by the luminance variancevalue calculating device.
 2. The abnormality detecting apparatus for animaging apparatus according to claim 1, further comprising: a regionexcluding device that sets an excluded region outside the scope ofexecution of the process to calculate the luminance variance value bythe luminance variance value calculating device in the region set by theregion setting device, wherein the luminance variance value calculatingdevice calculates the luminance variance value for the region other thanthe excluded region set by the region excluding device among the regionset by the region setting device.
 3. The abnormality detecting apparatusfor an imaging apparatus according to claim 2, wherein the regionexcluding device sets as the excluded region a region in which theluminance value calculated by the luminance value calculating device isnot less than a specified luminance value.
 4. The abnormality detectingapparatus for an imaging apparatus according to claim 2, furthercomprising: an object extraction device that extracts an object from theimage; and an object type determining device that determines a type ofthe object extracted by the object extraction device, wherein the regionexcluding device sets the excluded region based on a determinationresult of the object type determining device.
 5. The abnormalitydetecting apparatus for an imaging apparatus according to claim 4,wherein the region excluding device sets as the excluded region a regionon the image where an object determined to be a type of object with arelatively high luminance exists in the determination result of theobject type determining device.
 6. The abnormality detecting apparatusfor an imaging apparatus according to claim 1, further comprising: aluminance variance average value calculating device that calculates anaverage value of a plurality of the luminance variance values calculatedover a plurality of times by the luminance variance value calculatingdevice, wherein the abnormality detecting apparatus detects the presenceof abnormalities in the imaging apparatus in accordance with whether ornot the average value calculated by the luminance variance average valuecalculating device exceeds a predetermined threshold value.
 7. Theabnormality detecting apparatus for an imaging apparatus according toclaim 6, further comprising: a vehicle status quantity detection devicethat detects a status quantity of the vehicle on which the imagingapparatus is mounted, wherein the luminance variance average valuecalculating device sets the average value based only on the luminancevariance value calculated by the luminance variance value calculatingdevice in the state of the status quantity of the vehicle detected bythe vehicle status quantity detection device being a prescribed statusquantity.